Data Quality Adapters

Data Quality ensures that the information used to make key business decisions is reliable, accurate, and complete. It is therefore crucial to ensure data quality throughout the data management process.

• Data Quality stands as a critical factor in guaranteeing the integrity of information used for pivotal business choices, necessitating reliability, precision, and comprehensiveness. Consequently, upholding data quality across the entire data management continuum is of paramount importance.

• This metric assesses the extent to which a dataset satisfies specific benchmarks, gauging its alignment with predetermined standards.

• By ensuring data quality, a robust standard is maintained throughout data processing, fostering the utilization of data that is both accurate and dependable..

Ensuring the excellence of data quality encompasses a series of pivotal considerations and methodologies. The journey towards optimal data quality entails the following sequential steps:

• Data Profiling - Looks closely at where your data comes from to see how it's structured and how good it is. This helps catch problems before they cause trouble later.

• Data Cleansing - Removing or correcting errors, inconsistencies, and inaccuracies in the data. This may involve processes like deduplication, standardization, and validation.

• Data Transformation Rules - Applying business rules and transformations to ensure that data is properly formatted, normalized, and conforms to the target schema.

• Data Validation - Checking the transformed data to ensure that it meets predefined quality criteria and business rules.

• Data Enrichment - Enhancing data with additional information from external sources to improve its completeness and relevance. For example, handling null and empty values.

• Error Handling - Establishing mechanisms to identify and address data quality issues during the data processing.

Poor quality of data can lead to incorrect insights, inaccurate analysis, and unreliable business decisions. Organizations therefore invest significant effort in establishing robust data quality practices.

APIs and Interfaces used in Data Quality

processDataAnalyser(JobRunStatus jobRunStatus, DataStoreBean target, Map<String, Object> targetAttrMap, Map<String, Object> sourceAttrMap) 

switch (constraint.getConstraintName()) { 
 case "MaxLength": 
 applyConstraint("max(length(%s)) as %s,", constraint.getColumn1(), constraint.getCustomColumnName(), "", columnQuery); 
 break; 
} 

DataFrame handleOutlierData(DataFrame df, Map<String, List<Outlier>> outlierData) 
switch (action) { 
 case "max": 
 Object max = (cloneDf.select(Functions.max(Functions.col(columnName))).collect()[0].get(0)); 
 df = df.withColumn(columnName, Functions.when(exp, Functions.lit(max.toString())).otherwise(Functions.col(columnName))); 
 break; 
}

processDataProfiler(JobRunStatus jobRunStatus, DataStoreBean target, Map<String, Object> targetAttrMap, Map<String, Object> sourceAttrMap) 

switch (profilerFeature) { 
 case "characterCount": 
 pf.calculateCharCounts(session, tableName, columnList); 
 break; 
}

Following are the APIs and interfaces used in data quality:

• CalculateCompleteness()

  • This method takes tableName and profiler as a parameter.

  • It calculates completeness profiler result data.

• ProcessDataProfiler()

  • This method takes SourceDataStore and TargetDataStore bean as a request parameter.

  • It processes and generates profiler output.

  • It returns profiler result set count.

• HandleMissingData ()

  • This method takes dataframe, dropNullValues and replaceNullValues as parameter.

  • It handles and replaces missing data.

  • Returns result set data frame of issue resolver.

• HandleOutlierData ()

  • This Issue Resolver method takes dataframe and outlierData list as a paramter.

  • It handles outliers such as drop, mean, max and min etc.

  • It returns handled outlier data result dataframe.

• HandleCaseSensitiveData ()

  • This method takes dataframe and caseSensitiveData as parameter.

  • It handles data with functions such as upper, proper, lower etc.

  • Returns result set data frame.

• AnalyserandValidator

  • This method takes dataframe, constraints and table name as a request parameter.

  • It handles data with constraints such as Size, ApproxDistinctCount, MinLength, MaxLength, CountDistinct, Sum, Mean, StandardDiviation, Correlation, Completeness, Compliance, Uniqueness, Entropy, isLessThan, isGreaterThan etc.

  • Returns AnalyserAndValidator output.

Data Quality constraints

Based on the exact use case, data quality constraints or rules can be added. The supported constraints are as follows: